|
|
Add 8. run cts -m CtsJniTestCases -t android.jni.cts.JniStaticTest#test_linker_namespaces This item crash during CTS 09-29 14:12:11.414 9150 9178 F libc : Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 9178 (AndroidTestSuit), pid 9150 (android.jni.cts) 09-29 14:12:11.473 9183 9183 I crash_dump32: obtaining output fd from tombstoned, type: kDebuggerdTombstone 09-29 14:12:11.474 3597 3597 I /system/bin/tombstoned: received crash request for pid 9178 09-29 14:12:11.475 9183 9183 I crash_dump32: performing dump of process 9150 (target tid = 9178) 09-29 14:12:11.494 9183 9183 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** 09-29 14:12:11.495 9183 9183 F DEBUG : Build fingerprint: 'SKB/BFX_AT100/BFX-AT100:10/QTT5.200429.010/15.522.3-20200925:user/release-keys' 09-29 14:12:11.495 9183 9183 F DEBUG : Revision: '0' 09-29 14:12:11.495 9183 9183 F DEBUG : ABI: 'arm' 09-29 14:12:11.495 9183 9183 F DEBUG : Timestamp: 2020-09-29 14:12:11+0800 09-29 14:12:11.495 9183 9183 F DEBUG : pid: 9150, tid: 9178, name: AndroidTestSuit >>> android.jni.cts <<< 09-29 14:12:11.495 9183 9183 F DEBUG : uid: 10072 09-29 14:12:11.495 9183 9183 F DEBUG : signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr -------- 09-29 14:12:11.495 9183 9183 F DEBUG : Abort message: 'No pending exception expected: java.lang.ClassNotFoundException: com.skb.btv.framework.BtvJniInterface 09-29 14:12:11.495 9183 9183 F DEBUG : at java.lang.String java.lang.Runtime.nativeLoad(java.lang.String, java.lang.ClassLoader, java.lang.Class) (Runtime.java:-2) 09-29 14:12:11.495 9183 9183 F DEBUG : at java.lang.String java.lang.Runtime.nativeLoad(java.lang.String, java.lang.ClassLoader) (Runtime.java:1115) 09-29 14:12:11.495 9183 9183 F DEBUG : at void java.lang.Runtime.load0(java.lang.Class, java.lang.String) (Runtime.java:936) 09-29 14:12:11.495 9183 9183 F DEBUG : at void java.lang.System.load(java.lang.String) (System.java:1631) 09-29 14:12:11.495 9183 9183 F DEBUG : at java.lang.String android.jni.cts.LinkerNamespacesHelper.loadWithSystemLoad(java.lang.String) (LinkerNamespacesHelper.java:258) 09-29 14:12:11.495 9183 9183 F DEBUG : at java.lang.String android.jni.cts.LinkerNamespacesHelper.runAccessibilityTestImpl(java.lang.String[], java.lang.String[], java.lang.String[], java.lang.String[]) (LinkerNamespacesHelper.java:-2) 09-29 14:12:11.495 9183 9183 F DEBUG : at java.lang.String android.jni.cts.LinkerNamespacesHelper.runAccessibilityTest() (LinkerNamespacesHelper.java:204) 09-29 14:12:11.495 9183 9183 F DEBUG : at void android.jni.cts.JniStaticTest.test_linker_namespaces() (JniStaticTest.java:41) 09-29 14:12:11.495 9183 9183 F DEBUG : at java.lang.Object java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) (Method.java:-2) 09-29 14:12:11.495 9183 9183 F DEBUG : at void junit.framework.TestCase.runTest() (TestCase.java:168) 09-29 14:12:11.495 9183 9183 F DEBUG : at void android.jni.cts.JniTestCase.runTest() (JniTestCase.java:33) 09-29 14:12:11.495 9183 9183 F DEBUG : at void junit.framework.TestCase.runBare() (TestCase.java:134) 09-29 14:12:11.495 9183 9183 F DEBUG : at void junit.framework.TestResult$1.protect() (TestResult.java:115) 09-29 14:12:11.495 9183 9183 F DEBUG : at void androidx.test.internal.runner.junit3.AndroidTestResult.runProtected(junit.framework.Test, junit.framework.Protectable) (AndroidTestResult.java:73) 09-29 14:12:11.495 9183 9183 F DEBUG : at void junit.framework.TestResult.run(junit.framework.TestCase) (TestResult.java:118) 09-29 14:12:11.495 9183 9183 F DEBUG : at void androidx.test.internal.runner.junit3.AndroidTestResult.run(junit.framework.TestCase) (AndroidTestResult.java:51) 09-29 14:12:11.495 9183 9183 F DEBUG : at void junit.framework.TestCase.run(junit.framework.TestResult) (TestCase.java:124) 09-29 14:12:11.495 9183 9183 F DEBUG : at void androidx.test.internal.runner.junit3.NonLeakyTestSuite$NonLeakyTest.run(junit.framework.TestResult) (NonLeakyTestSuite.java:62) 09-29 14:12:11.495 9183 9183 F DEBUG : at void androidx.test.internal.runner.junit3.AndroidTestSuite$2.run() (AndroidTestSuite.java:101) 09-29 14:12:11.495 9183 9183 F DEBUG : at java.lang.Object java.util.concurrent.Executors$RunnableAdapter.call() (Executors.java:462) 09-29 14:12:11.495 9183 9183 F DEBUG : at void java.util.concurrent.FutureTask.run() (FutureTask.java:266) 09-29 14:12:11.495 9183 9183 F DEBUG : at void java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) (ThreadPoolExecutor.java:1167) 09-29 14:12:11.495 9183 9183 F DEBUG : at void java.util.concurrent.ThreadPoolExecutor$Worker.run() (ThreadPoolExecutor.java:641) 09-29 14:12:11.495 9183 9183 F DEBUG : at void java.lang.Thread.run() (Thread.java:919) |
|
|
hi jason.liang per talking to you in webchat. please check it latest version ( today will build it. please give a result in next week. when can i receive your result in next week ? |
|
|
Hi Mr. Jason Park, Do you want to run "full" CTS, including manual Testing, if so, I need to arrange our QA resource, and the "full" test result will generated on next Friday. Please also ask our engineer to release "SU" build for testing. |
|
|
hi Jason.liang yes. we need to test full,since other company also tested with v15.522.9 version. Thanks. |
|
|
Hi Mr. Jason Park, Have you forwarded these failures to Sptek, Our QA are testing v15.522.10 this week, we have finished some test item, but I still see these failures. |
|
|
hi jaosn.liang we informed it before. but after finish it with v15.522.10, please share it. Does you need to build SU image? as you know it, we build it everyday. Frankly speaking, it isn't easy to build SU Image. if you give a plan, I wll make the image as SU. since AML patch release everyday, so give your plan which version is used for CTS full test. this step is for checking a current level. after fw is stabled, we have to test it again. |
|
|
Add 9. run cts -m CtsSecurityHostTestCases -t android.security.cts.SELinuxHostTest#testNoBugreportDenials junit.framework.AssertionFailedError: Found illegal SELinux denial(s): avc: denied { getattr } for path="/btv_home" dev="mmcblk0p19" ino=2 scontext=u:r:dumpstate:s0 tcontext=u:object_r:btv_home_file:s0 tclass=dir permissive=0 avc: denied { search } for name="/" dev="mmcblk0p19" ino=2 scontext=u:r:dumpstate:s0 tcontext=u:object_r:btv_home_file:s0 tclass=dir permissive=0 avc: denied { search } for name="/" dev="mmcblk0p19" ino=2 scontext=u:r:dumpstate:s0 tcontext=u:object_r:btv_home_file:s0 tclass=dir permissive=0 at junit.framework.Assert.fail(Assert.java:57) at junit.framework.Assert.assertTrue(Assert.java:22) at junit.framework.TestCase.assertTrue(TestCase.java:192) at android.security.cts.SELinuxHostTest.testNoBugreportDenials(SELinuxHostTest.java:970) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at junit.framework.TestCase.runTest(TestCase.java:176) at junit.framework.TestCase.runBare(TestCase.java:141) at com.android.tradefed.testtype.DeviceTestResult$1.protect(DeviceTestResult.java:90) at com.android.tradefed.testtype.DeviceTestResult.runProtected(DeviceTestResult.java:65) at com.android.tradefed.testtype.DeviceTestResult.run(DeviceTestResult.java:94) at junit.framework.TestCase.run(TestCase.java:129) at com.android.tradefed.testtype.DeviceTestCase.run(DeviceTestCase.java:181) at com.android.tradefed.testtype.JUnitRunUtil.runTest(JUnitRunUtil.java:53) at com.android.tradefed.testtype.JUnitRunUtil.runTest(JUnitRunUtil.java:37) at com.android.tradefed.testtype.DeviceTestCase.run(DeviceTestCase.java:146) at com.android.tradefed.testtype.HostTest.runRemoteTest(HostTest.java:608) at com.android.tradefed.testtype.HostTest.runTestClasses(HostTest.java:535) at com.android.tradefed.testtype.HostTest.run(HostTest.java:525) at com.android.compatibility.common.tradefed.testtype.JarHostTest.run(JarHostTest.java:53) at com.android.tradefed.testtype.suite.GranularRetriableTestWrapper.intraModuleRun(GranularRetriableTestWrapper.java:413) at com.android.tradefed.testtype.suite.GranularRetriableTestWrapper.run(GranularRetriableTestWrapper.java:229) at com.android.tradefed.testtype.suite.ModuleDefinition.run(ModuleDefinition.java:466) at com.android.tradefed.testtype.suite.ITestSuite.runSingleModule(ITestSuite.java:691) at com.android.tradefed.testtype.suite.ITestSuite.run(ITestSuite.java:592) at com.android.tradefed.invoker.InvocationExecution.runTests(InvocationExecution.java:480) at com.android.tradefed.invoker.TestInvocation.prepareAndRun(TestInvocation.java:380) at com.android.tradefed.invoker.TestInvocation.performInvocation(TestInvocation.java:230) at com.android.tradefed.invoker.TestInvocation.invoke(TestInvocation.java:754) at com.android.tradefed.command.CommandScheduler$InvocationThread.run(CommandScheduler.java:594) |
|
|
Add 10, 11, 12 |
|
|
Hi Mr. Jason Park Please check each sheet of the attachment (BTF_15.520.10.xlsx) I use different color to represent who should fix the failure. White: Known issue (Already waived by Google) Green: Foxconn Orange: Amlogic Blue: SPtek We have reported Amlogic's issue to Mega, I also update this post for Sptek's issue. Please inform the issues in these post to Sptek. |
|
|
Hi Mr. Jason Park Update excel file (BTF_15.520.10_20201026.xlsx) Add BTS sheet in excel, this test is for malware scanning, we just upload image file to Google server, the server will scan the malware and unsafe api/function in app Add 13,14,15,16 |
|
|
HI Jason.liang according to your CTS test via v15.522.10, I received some question from Digicap regarding to browser part violation. please check their comment. Digicap don't know about NUGU usage. however, other part(HomeUI/Zemkids/Senior) usage is shown as below. . promotion pop-up. . Pop-up for broken for monthly fee. . TVpoint and TVPay . change password. . STB Upgrade. . etc.. according to sever form guide, it may use a several method. it need to proceed it in each web server... They wonder it hope receive google waiver. |
|
|
Hi Mr. Jason Park, Based on our past experience, we haven't get any BTS waiver from Google. In the past, we also have BTS fail on other project. at that time, we checked with 3PL "Have other OEM ever got BTS wavier before?" They said "never", so we fix it. If Digicap still want to request waiver, I think the Digicap should provide the source code for Google reivewing and discuss it with Google. |
|
|
hi Jason.liang Thanks for your feedback. I will toss it to SKB Manager and Digicap. today we release v15.522.22. please start for Google test with this version. |
|
|
Hi Mr. Jason Park, Our QA has started Test v15.522.20 this morning. if you still want to test v15.522.22. They will start test on next Monday, because they are out of office for business trip. |
|
|
hi Jason.liang. digicap ask to us about an workaround solution of BTS fail item. all application have to be included a certification of authentication for passing google requirement, however it isn't easy job. so digicap want to guide for passing . as below link, when user enter website, make a step for asking some warning message " it need a certification of authentication " Does it can accept this for pass ? or if you know a method to pass for BTS fail, please let me know a guide. https://support.google.com/faqs/answer/7071387?hl=en th |
|
|
Hi Mr. Jason Park, I am not an application developer, so I don't know how to guide Digicap member to pass this item. I'll forward this question to our application engineer. Could Digicap provide more information about implementation of the WebViewClient.onReceivedSslError handler ? Hi Rex, Could you check Digicap's request? |
|
|
Hi Mr. Jason Park, We have encountered the BTS failed before. Google does not allow developer to proceed the SSL certification without any check or block in the WebViewClient.onReceivedSslError method. The official method is to check the available of SSL certification then judgment the request is proceed or cancel. Or we can reference the browser’s flow, prompting the user that the certification was illegal, and let the user decide whether to proceed or cancel. |
|
|
Hi Mr. Jason Park, Regarding the v15.522.22, the BTS (maleware scanning) is in progress, I will update the excel sheet later when we finishing BTS. Remove 1,3,5,11,14,16 (fix in v15.522.22) |
|
|
Hi Mr. Jason Park, Update v15.522.22 result |
|
|
hi Jason.liang. SPtek ask to us give a log about Sptek's fail all item in excel sheet. example)… Amlogic ref board/image can pass this item without connecting HDMI , I have tried to remove HDMI cable and test our image, but still fail, please check this item with Sptek. k000b-r01g_SD-20200707 can pass |
|
|
Hi Mr. Jason Park CTS is public test suite, everyone can download it. I have uploaded the GTS/VTS log to you. Please also check smoke test sheet, they can verify it on their side. |
|
|
hi Jason.liang please test with v15.522.26 for google certification TEST within This week. |
|
|
Hi Jason P., How about the progress of failed items for M/W ? I am afraid that we don't have much time to do pre-testing if we'd like to submit the final F/W on 11/18. |
|
|
hi Jason. SPTek try to solve it within this week. You already start for Google test with v15.522.26 ? when we receive a result again ? I received some comment from SPTek and Digicap. according to your test result, it is a difference point, Digicap applied xwalk in v15.522.22 version. 10/26 - HomeUI , ZemKids, Senior, Nugu. ( before apply xwalk) 11/04 - Zemkids, Senior - (after Apply xalk) Thanks. 11_04.JPG (60,193 bytes) 10_26.JPG (98,010 bytes) |
|
|
Hi Mr. Jason Park, We had started testing v15.522.26 this Monday. All results will be generated on this Friday, In this meantime, if we find any new failures, I will let you know. Regarding the BTS failures, I don't know what xwalk is. The BTS test suite is on Google Cloud. Google frequently changes the test suite, We cannot specify which version to use for testing. We don't have source code of BTS test suite, so we don't know the change list of BTS test suite. Google use BTS (version 2020-11-05T09:00:06.196Z) to scan our FW v15.522.22 Google use BTS (version 2020-10-27T13:52:03.779Z) to scan our FW v15.522.10 |
|
|
Hi Mr. Jason Park Could you check one cts fail with Intek or Innopia, the test item name is "android.server.wm.MultiDisplayClientTests#testDisplayIdUpdateOnMove_NoRelaunchActivity". the original owner of this issue is Amlogic, but they verified on their reference board, they don't have this issue, they think this issue may from other Vendor. Could you help us check with other OEM first ? |
|
|
Hi Mr. Jason Park, We have finished the auto test of v15.522.26, the remaining tests (BTS/SmokeTest/Youtube Test) are in progress. We found that the number of failures has increased, Please check the CTS and CTS-ON-GSI sheet, I use gray color to highlight the new failures that are different from the v15.522.22. if other OEM also find these new failures, Please check with Sptek first. |
|
|
hi Jason.liang SPTtek also still working to resolve about google test items. please refer to an attached file from SPtek. they will resolved it until this week. and we consider to extend a booking date at next Tuesday.. |
|
|
Hi Mr. Jason Park Update CTS/CTS-ON-GSI sheet, Add BTS/SmokeTest sheet, Youtube Test is in progress. Change the owner of issue "android.server.wm.MultiDisplayClientTests#testDisplayIdUpdateOnMove_NoRelaunchActivity" from Amlogic to Sptek, because Amlogic pure new sdk with their reference board doesn't have this failure. Update BTS sheet, Does UI vendor fix it on v15.522.26, because we still have these failures. Update SmokeTest Sheet Play Movies with 5.1 Audio (new fail), so we can not verify the behavior of volume bar. |
|
|
HI Jason.liang please if you have some result of Google test with V15.22.27, please share it. or if you want to check a current status with using build by yourself, you can do it. I will ask to build new FW today for supporting about BMT issue. |
|
|
Hi Mr. Jason Park, You haven't ask us to Test 15.522.27, so we don't have any report of 15.522.27. I will verify the current failure this afternoon with latest code. |
|
2020-11-17 18:11 developer |
|
|
|
Hi Mr Jason Park, I have built the latest code this afternoon and verified the current failures, I have removed the item that already fix in latest code, please keep checking the blue color item in each sheet. |
|
2020-11-17 18:26 developer |
|
|
|
Hi Mr. Jason Park, Regarding item 19, the result from our QA is fail, but Sptek said they don't encounter this issue. I rollback to verify 15.522.22 again, at that time, the report from our QA is pass, but I verify it with 15.522.22 yesterday, the pass rate is too low. I also verify this item on Amlogic ref board, the result is PASS. I check this question with Amlogic, they said that there is only one customer report this issue in their internal Jira. and the issue was happen from customer UI. if Sptek do not encounter this issue, please let us know how to pass it, any special step ? In addition, I try to replace our system partition with Google System Image (standard system.img) and test this item again, it's easy pass (5 test, 5 pass). so I guess maybe our framework has some background process or application impact this item. |
|
|
Hi Mr. Jason Park the BTS result of yesterday noon's latest code is clean. Remove 14, 15. ========================================================= Sorry, the result of BTS of 15.522.27 with latest code hasn't been generated. We are waiting for it. Anyway, I think Digcap is fixing it . |
|
|
hi Jason. so you didn't check all fail item in BTF_15.520.27_latestCode-20201117_noon_byFoxconn-2.xlsx. some part need to updated for a result ? |
|
|
Hi Mr. Jason Park, I have checked, the BTF_15.520.27_latestCode-20201117_noon_byFoxconn-2.xlsx is the new result verified with 15.52027 with latest code. I have removed some item that already fix by Sptek, you can compare between BTF_15.520.27_latestCode-20201117_noon_byFoxconn-2.xlsx and BTF_15.520.26-20201112.xlsx |
|
|
Hi Mr. Jason Park, the BTS result of 15.522.27 with latest code is generated, please check the picture, Please check these 3 items Please don't care about the version, because the firmware was build by myself, for test only. Reopen item 14,15,16 bts-20201117.jpg (227,736 bytes) |
|
|
Hi Mr. Jason Park, Did other OEM verify 15.522.27 ? A few days ago, Google release new CTS tool (10_r6), so our QA use 15.522.27 to verify new CTS tool. Our QA find new failures in 15.522.27 with the new CTS tool (10_r6), In the meantime, I also use latest code (2020/11/18 night) to verify these failures with "current" tool (10_r5) Please check the following new failures, maybe someone make side effect. I use the latest code with CTS 10_R5, these failures still happen. Add item 20, 21, 22 in main post run cts -m CtsAppTestCases -t android.app.cts.InstrumentationTest#testSendStringSync run cts -m CtsViewTestCases -t android.view.cts.ViewTest#testUnhandledKeys run cts -m CtsWidgetTestCases -t android.widget.cts.AutoCompleteTextViewTest#testOnFilterComplete run cts -m CtsMediaTestCases -t android.media.cts.VideoDecoderPerfTest#testVp9Other0Perf0320x0180 (This belongs Amlogic, I have informed them) |
|
|
hi Jason.liang I receive some question from SKB manager about a screen saver("Dreams") SKB think it only possible to enable to work in google apps except SKB apps. so skb manager asked to us which apps of Googles have to do for Dreams test item. Let me know a list of apps. Is there any problem excluding the SKB app? after SKB apps launch , a screen saver don't work it. Dreams.JPG (59,432 bytes) |
|
|
Hi Mr. Jason Park, In my opinion, I think all apps should be able to enter dream mode, unless the app is playing videos or games. I ask this item with 3PL tester, their opinions are the same as mine. I think I have told you before, Google TAM is more rigorous in SW reviewing from this year. I also know that some of Korean Google TAM use Smart3 in their home. Please fix it. |
|
|
hi Jason Thanks . I will inform it. you know a list of the app of Google. fist we need to check this app list. except SKB apps. |
|
|
Hi Mr. Jason Park, Update the test result of version 15.522.28b on 2020/11/23 CTS: In progress CTS-ON-GSI: In progress SmokeTest: In progress. YoutubeTest: in progress. CTS-verifier: Finished, result is clean STS: Finished, result is clean. VTS: Finished, result is clean. TVTS: Finished, result is clean. GTS: FInished, Item 4 has not been fix. 4. run gts -m GtsSecurityHostTestCases -t com.google.android.security.gts.SELinuxHostTest#testNoExemptionsForBinderInVendorBan (Reason: Policy exempts vendor domains from ban on Binder: [btvservice_hal, ctcservice]) BTS: Finished, Item 14,15,16 has not been fix. 14. BTS::MALWARE_FINDER#SYSTEM//system/app/SKB_Senior_SMART3BTF_foxconn/SKB_Senior_SMART3BTF_foxconn.apk | DeepDive (The following issue has been detected in the APK com.skb.senior, which you should investigate before launching this build: Your app is using an unsafe implementation of the WebViewClient.onReceivedSslError handler. Vulnerable Classes: com.skb.senior.popupFragment.browser.BrowserPopupFragment$BtvSTBWebViewClient) 15. BTS::MALWARE_FINDER#SYSTEM//system/app/SKB_ZemKids_SMART3BTF_foxconn/SKB_ZemKids_SMART3BTF_foxconn.apk | DeepDive (The following issue has been detected in the APK com.skb.zemkids, which you should investigate before launching this build: Your app is using an unsafe implementation of the WebViewClient.onReceivedSslError handler. Vulnerable Classes: com.skb.zemkids.popupFragment.browser.BrowserPopupFragment$BtvSTBWebViewClient) 16. BTS::MALWARE_FINDER#SYSTEM//system/app/nugu/nugu.apk | DeepDive (The following issue has been detected in the APK com.skt.aidev.nugustb, which you should investigate before launching this build: Your app is using an unsafe implementation of the WebViewClient.onReceivedSslError handler. Vulnerable Classes: b.a.b.a.b.d1$b) |
|
|
hi Jason.liang please test with v15.522.29 about Google test. give a result within today . it is ok for a result of auto (simple) we will decide whether if a current FW will made new fw for google certification or not based on your report. |
|
|
Hi Mr. Jason Park, You ask us to test v15.522.28b last Friday. The test has not been finished. Do you want us to stop testing 15.522.28b? I don't know that what is "give a result within today". All test suite take 5 days to complete. Did you mean we only verify the current known failure and give you the result of current failure? |
|
|
hi Jason.liang. SPtek will make a new FW which is solved for Google issue. so please stop your test with before fw. we will give a new FW. Please use it. During SKB today meeting, we have some question regarding to Google certification test of android q. 1. other vendor met CTS fail via google launcher. we disable a google launcher in android 9. but There is two launcher (google launcher and SKB launcher) in android 10. although it set to adjust skb launcher as top priority, if SKB launcher is killed, it only remain Google launcher. so it is a risk point in SKB biz. please search it in google test. this feature is mandatory item in Android 10 ? when enable for a google launcher during a google test, we can modify to disable it later. is it possible to do this? how can we pass this item ? or it is impossible to do this... how can we make an workaround solution? see googlelauncher.png 2. What is a problem about below issue? this problem is related to MW? when it is used same MW in SWAN project, there is no problem for this issue. so SPtek wants to know which problem for this issue? see CtsWindorManagerDeviceTestCases.JPG. 3. SPTEK don't meet a failure below Test. let me know how can we test for this item? if you give it,, I will compare to SPTEK'S test procedure. if you want to know sptek test procedure, please I will ask it to SPTek for you. see CtsApptestCase.JPG. googlelauncher.png (15,046 bytes) CtsWindorManagerDeviceTestCases.JPG (28,352 bytes) CtsApptestCase.JPG (31,022 bytes) |
|
|
Hi Mr. Jason Park, 1. other vendor met CTS fail via google launcher. Actually, We don't "disable" Google launcher in the firmware. we remove TVLauncher (Google Launcher) from GTVS built-list, our firmware do not contain the Google Launcher. So we don't need to "disable" TVLauncher during testing. I don't know why other OEM prebuilt the TVLauncher. The eagle project is operator tier device, and SKB use customer launcher. I think other OEM can check the document on ATV Help, The TVLauncher is not required when customer use a customer launcher. BFX-AT100 only contain one launcher, so we don't have the CTS issues that related to multi-launcher. =========================================================================================================== 2. What is a problem about below issue? Regarding this case "CtsWindorManagerDeviceTestCases", I don't know who is the real owner is. At first, I assigned this issue to Amlogic, but Amlgoic asked us to verify it with their standard SDK first. I have verified this item with Amlogic standard SDK on their ref board at least 5 times, all results are pass. It's stable. so I change the owner to MW. I also ask you to check this item with other OEM, other OEM also have this item. Does other OEM have any idea on this issue? Regarding the swan's result provided by MW. I also check it before. Can Sptek confirm all MW/3rd party apps are same between swan and eagle ? There is another possibility that Amlogic specially helped the SKB to change something, that modification did not exist on their standard SDK, so their standard SDK can pass. the result of ours is fail =========================================================================================================== 3. CtsApptestCase, the following result are from our QA 15.522.10 Our QA didn't encounter it. 15.522.22 Our QA didn't encounter it. 15.522.26 Our QA encounter it, and they retried many times, the result is still fail. 15.522.27 Our QA didn't encounter it. Based on the above results, I went back to re-test 15.522.22, and I found the pass rate of this item was very low. I "guess" this test may be affected by background services, so the result is random. because Amlogic standard SDK do not have this issue. (I verified it on ref board at least 5 times, all resutls are pass) I am very worried about how to overcome this problem when we encounter this problem during formal certification, so I highlight it. |
|
|
Hi Mr. Jason Park, Could you check spec.jpg with Digcap. our QA member said when BOX is online, We did not meet any of the following conditions 1. Recency 2. Alphabetical order 3. Order edited by user When box is offline, we can see the blue bottom on top-right corner. (order.jpg) we can press blue-button and set Alphabetical order, when box back to online, the result is good. But when we reboot again, after box boot completed with online. We did not meet any of the following conditions again. 1. Recency 2. Alphabetical order 3. Order edited by user spec.jpg (76,012 bytes) order.jpg (124,800 bytes) |
|
|
Hi Mr. Jason Park, Regarding Dream mode issue, we can enter dream mode when we stay at TvSettings (Google Settings) You ask us test the behavior of Google apps, Our QA verify the behavior on other Android TV. (Playstore/Youtube/Youtube Music/PlayMovie/PlayGame) can enter dream mode when the device is inactive on other Android TV. But Smart3 can not enter dream mode in Google apps when device is inactive. Please check it. |
|
|
hi Jason.liang. SPtek said to me this decision is confirmed by SKB user... so they only enter in tvsetting. Smart 3 in your comment is android 10 Eagle ? |
|
|
Hi Mr. Jason Park SPtek said to me this decision is confirmed by SKB user... so they only enter in tvsetting. OK, If Google or 3PL ask us about this question, I'll use this comment to reply to them, but I am not sure if they can accept it. Smart 3 in your comment is android 10 Eagle ? yes, BTF (Android 10, eagle) can not enter dream mode in Google apps when device is inactive. |
|
|
hi Jason regarding to test fail, Intek box is ok by SPtek test. so sptek want to check all apps by FXN. They will check it.. give a apk list,, they check one by one in list of FXN about this tests. although intek also have an apps, SPtek do something for intek's apk. they want to check it |
|
2020-11-25 16:31 developer |
CtsWindorManagerDeviceTestCases-2.JPG (28,352 bytes) |
|
|
Hi Jason, Park Have Sptek also verified CTS-ON-GSI of this item (CtsWindorManagerDeviceTestCases) with intek Box? |
|
|
hi Jason.liang SPtek will test it again. per talking in wechat,, plz start with v15.522.31. |
|
|
Hi Mr. Jason Park, Regarding item 8, run cts -m CtsJniTestCases -t android.jni.cts.JniStaticTest#test_linker_namespaces it's still fail on version r31, the library if from Digicap, library path (vendor/skb/prebuilts/digicap_btv-app/libglide-webp.so) |
|
|
Hi Mr. Jason Park, Our QA said the order of apps is weird. Test Version 15.522.31 We have known the "out-of-the-box" order of Google app Must be 1. Play Store, 2. Play Movies & TV, 3. YouTube, 4. YouTube Music, and 5. Play Games, And we also need to follow one of following condition of app order. 1. Recency 2. Alphabetical order 3. Order edited by user Our QA disconnect the box, and press blue button, choose "Alphabetical order" Please see the offline Picture (15.522.31_app_offline.jpg) The order of app in offline is as below BADLAND -> Google Play Store -> Google Play Movie -> Google Play Game -> RED BALL -> Smart Home -> Youtube -> Youtube Music -> Youtube Kid, Zeus. Seems to meet the requirement. but Google Play 'S'tore -> Google Play 'M'ovie -> Google Play 'G'ame is incorrect, it should be Google Play 'G'ame -> Google Play 'M'ovie -> Google Play 'S'tore ( G -> M -> S) Youtube -> Youtube Kid -> Youtube Music (K -> M) Even more strange is when smart3 go to online. (15.522.31_app_online.jpg) The first five positions are still fixed, The remaining apps are follow Alphabetical order. 1. Online and offline are inconsistent. (the first five positions are always fix when box is online) 2. I think the first five positions are fixed and only apply for "out of the box", when user change the order, we don't need fix first five position. Please check it with the Digcap. 1-15.522.31_app_offline.jpg (69,965 bytes) 2-15.522.31_app_online.jpg (130,067 bytes) |
|
|
hi Jason.liang. sptek committed some solution for CtsWindowMangerDeivicesTestCase.. case intek , it is passed. however, it is failed for CTS-ON-GSI of this item (CtsWindorManagerDeviceTestCases) with intek box. Could you check to build by internally ? please let me know it. according to pass for CtsWindowMangerDeivicesTestCase.., SPtek think they isn't an owner. is it right? if they isn't an owner, who is an owner for CTS-ON-GSI of this item (CtsWindorManagerDeviceTestCases) ? by amlogic…. |
|
|
Hi Mr. Jason Park, Regarding the CtsWindorManagerDeviceTestCases failure. I go back to re-test 15.522.22 the result of CTS and CTS-on-GSI are pass. I also use the latest image 15.522.31 the result of CTS and CTS-on-GSI are pass. Currently, I don't know why, Please check the attachment. |
|
|
Hi Mr. Jason Park, Daily update failure on 15.522.31 1. the PlayMovie is no sound when Smart3 in PCM mode. (Already report to Amlogic, they are working with Sptek) 2. run cts -m CtsAppTestCases -t android.app.cts.AlarmManagerTest#testAlarmTriggersImmediatelyIfSetTimeIsNegative (still difficult to pass) |
|
|
hi Jason.liang. as inform in Webchat, please test with v15.522.34.. |
|
|
Mr. Jason Park, Does Sptek fix run cts -m CtsAppTestCases -t android.app.cts.AlarmManagerTest#testAlarmTriggersImmediatelyIfSetTimeIsNegative I build the v15.522.34 by myself, it's still fail. |
|
|
Hi Mr. Jason Park, Remove 14,15,16, Add 23 in main post BTS::MALWARE_FINDER#SYSTEM//system/app/SmartHome/SmartHome.apk | DeepDive (Your app is using an unsafe implementation of the WebViewClient.onReceivedSslError handler. Vulnerable Classes: com.skt.tvsmarthome.ui.MainActivity$MainWebViewClient) I build 15.522.34t by myself (it's the same as 15.522.34), I submit this test image for malware scanning. Please check the smartHome.jpg |
|
2020-11-30 10:07 developer |
smartHome.jpg (131,002 bytes) |
|
|
hi Jason.liang Thank for your feedback. as your report, a new item(SmartHome) from Sptek on these days met BTS fail item. so I wonder since it already found fail item, Can't go into for 12_3 slot ? please give a guide to us. |
|
|
hi Jason.liang. Did you check below item in v15.522.34 later? 1. the PlayMovie is no sound when Smart3 in PCM mode. (Already report to Amlogic, they are working with Sptek) although I inform it to Sptek, they don't catch it. |
|
|
Hi Mr. Jason Park, >>>1. the PlayMovie is no sound when Smart3 in PCM mode. (Already report to Amlogic, they are working with Sptek) The PCM mode is work in 15.522.34 The app order is still fail even we change server configuration to stage. my step is change to stage server and fill v522 in the edit text, press save , then select prod button in item 6. (if my step is wrong, please correct me) Please check the following result. We select the sort by English alphabetic (item 3), the language settings of System is also English. the result in offline mode: (offline_v34.jpg) BADLAND -> Crossy Road -> Orbia _> Google Play Game -> Google Play Movie -> Google Play Store -> RED BALL -> Smart Home -> Youtube -> Youtube Kid -> Youtube Music _> Zeus Why the position "O"rbia is ahead of "G"oogle apps" the result in online mode: (online_v34.jpg) the first fix position is still fix, when we use sort by Alphabetical order. (I think the first five positions are fixed and only apply for "out of the box", when user change the order, we don't need fix first five position.) Did UI Vendor fix it ? Please check offline_v34 and online_v34. online_v34.jpg (137,317 bytes) offline_v34.jpg (86,305 bytes) |
|
|
I do the single test for the current known failure on v15.522.234 Remove item 4, 8, 10, 17, 18, 20, 21, Item10 is go to TvSetting for testing. Let me highlight the current known failure of v15.522.234, the other test suite is still in progress. 1. BTS::MALWARE_FINDER#SYSTEM//system/app/SmartHome/SmartHome.apk (item 23 in main post) 2. run cts -m CtsAppTestCases -t android.app.cts.AlarmManagerTest#testAlarmTriggersImmediatelyIfSetTimeIsNegative (item 19 in main post) This item can random pass, but the pass rate is very low, Does Sptek know the test method to pass it? 3. App order is incorrect. see comment 5158 and 5219 in this post (item 24 in main post) |
|
|
Hi Mr. Jason Park, Update Current Status (15.522.34) CTS: finished, Please check item 19, CTS-Verifier: finished. GTS: finished. CTS-ON-GSI: finished. STS: finished. VTS: finished. BTS: finished, 1 fail -> Smart Home apk issue (item 23) Smoke Test: in progress, but please check app order issue with UI Vendor first. Youtube Test: in progress |
|
|
Hi Mr. Jason Park, Could you check this issue, it is found when we test Smoke Test on (15.522.34) We can’t move the focus to the right. Please check the video (focus_issue.mp4) focus_issue.mp4 (12,629,678 bytes) |
|
|
hi Jason.liang I will ask it to skb manager about mantis 5236. 19. run cts -m CtsAppTestCases -t android.app.cts.AlarmManagerTest#testAlarmTriggersImmediatelyIfSetTimeIsNegative (pass rate is too low) sptek also try to investigate to find a reason about this issue. so if you have some idea, please share it with us. |
|
|
hi Jason.liang regarding to your comment at 5236, I checked it to skb manager. as your two case, it can't access it now. but skb webUI will update it today. so please follow up below step. 1. default is used as Prd server. 2. changing as STG for checking apps order. since until now , this item only apply on STB webUI server. --> please see change_STG.jpg. 3. after checking it, you need to change it as default. --> please see initialize.jpg 1) push hot key( *+7+8+9+6+#) in live channel. 2) push "적용" at #3 3) push "previous key" or "Home" key |
|
|
Hi Mr. Jason Park Add 25. "During the casting process, press the back key, the video is gone, but the audio remains in the background. Our QA had reported this bug in 15.522.22. You can check the attachment in comment 4886. At that time, we didn’t know how to reproduce it stably. And we haven’t found it in the past few versions, so we think this problem has been fixed. Now we can reproduce it. so I highlight it, this is related to audio/video, so I reported it to Amlogic first, You can also inform MW vendor, because as far as I know, MW vendor also modified the audio control. reproduce step: Open Chrome browser on PC >> Play Youtube video >> Click the 3 vertical dots on the top right corner of the chrome browser and select cast. Choose the correct device from the list to cast. In the process of streaming, press back key on RCU, then the video is gone. but the audio remains in the background. I also verify this issue on SWAN board. The Android version and version of chromecast app between EAGLE and SWAN are the same. But SWAN do not have this issue. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Mr. Jason Park, Please replace youtube with the following URL, it will be easier to reproduce. https://vimeo.com/channels/staffpicks/481737053 vimeo is the test item in smoke test sheet. |
|
|
Hi Mr. Jason Park, Add 26 (comment 5236) We have finished the pre test of version 15.522.34 Let me highlight the current known failure of v15.522.234 1. BTS::MALWARE_FINDER#SYSTEM//system/app/SmartHome/SmartHome.apk (item 23 in main post) 2. run cts -m CtsAppTestCases -t android.app.cts.AlarmManagerTest#testAlarmTriggersImmediatelyIfSetTimeIsNegative (item 19 in main post) This item can random pass, but the pass rate is very low, Does Sptek know the test method to pass it? (We do the factory reset at least 7 times, then we pass it) 3. App order is incorrect. see comment 5158 and 5219 in this post (item 24 in main post) 4. Focus sometimes can not go to right area (item 26 in main post (item 26 in main post) 5. During the chrome browser casting process, press the back key, the video is gone, but the audio remains in the background (item 25 in main post) the item 3~5 is found during smoketest, Item 3 is an obvious failure. regarding item4 and item 5, Because the judgment of smoketest is subjective, I cannot answer you whether this is just a bug or affects the certification. |
|
|
for 5245 comment , please see an attached picture. |
|
2020-12-02 10:01 developer |
|
|
|
hi Jason.liang we receive new tag from SPtek. please use with v15.522.38 for CTS Test. I heard this tag is included for revert HLS patch and apps order. 15.522.37 BFX-AT100_5.2.2_2020-1204-162814.xml 15.522.38 BFX-AT100_5.2.2_2020-1204-164641.xml |
|
|
Hi Mr. Jason Park, Update Current Status (15.522.38) 2020/12/07 morning CTS: In progress CTS-Verifier: In progress GTS: finished CTS-ON-GSI: In progress STS: In progress VTS: finished TVTS: finished BTS: finished Smoke Test: In progress Youtube Test: In progress Update Current Status (15.522.38) 2020/12/07 night CTS: In progress CTS-Verifier: finished GTS: finished CTS-ON-GSI: finished STS: finished VTS: finished TVTS: finished BTS: finished Smoke Test: In progress Youtube Test: In progress |
|
|
Update Current Status (15.522.38) 2020/12/08 morning CTS: In progress, retrying CTS-Verifier: finished GTS: finished CTS-ON-GSI: finished STS: finished VTS: finished TVTS: finished BTS: finished Smoke Test: In progress Youtube Test: finished Regarding version 38a, because we don't have enough time to run full test., so we test following item CTS: CtsMediaTestCases, finished TVTS: full, finished. GTS: full, finished. STS: full, finished. VTS: Omx Module, finished SmokeTest: Test screen mirror by Browser and Mobilephone |
|
|
Update Current Status (15.522.38) 2020/12/09 morning CTS: finished CTS-Verifier: finished GTS: finished CTS-ON-GSI: finished STS: finished VTS: finished TVTS: finished BTS: finished Smoke Test: In progress (target time: 12/9 noon) Youtube Test: finished |
|
|
Hi Mr. Jason Park, Remove 23,24,25,26 we have finished all test suite on v15.522.38. There is no issues with this version, except for known issue for falsely declared wifi feature. the wifi failure has already waived by Google. |
|
|
hi Jason.liang regarding to v15.522.38, give a result as document. since our sales team need this document at this moment. |
|
2020-12-09 15:28 developer |
|
|
|
Hi Mr. Jason Park, Please check the attachment. (BTF_15.520.38-20201204.xlsx) |
|
|
hi Jason.liang please use v15.522.42 version for google test .. I ask to build now. |
|
|
hi Jason.liang please give a result here with v15.522.42. I heard our test is finished until today. |
|
|
15.522.42 (2020.12.21 morning) CTS: In progress CTS-ON-GSI: In progress STS: In progress SmokeTest: In progress YoutubeTest: In progress CTS-Verifier: finished, clean VTS: finished, clean GTS: finished, clean TVTS: finished clean BTS: finsihed, clean |
|
|
Hi Mr. Jason Park, All tests are finished. the result is clean, please check the attachment. |
|
|
Hi Mr. Jason Park, Our QA have finished the pre-test of version 15.522.42c Please check the attachment. At this time, We don't test BTS, because 15.522.42 are reviewing by Google now, we don't upload the new build (15.522.42c) for malware scanning (if we upload 15.522.42c, it might confuse reviewer) Regarding SmokeTest and Youtube Test, we tested SmokeTest Lite version and skiped YoutubeTest this time. we assume 15.522.42 will be approved by Google, so next SW release will be MR. if the release is MR, we don't need to test YouTubeTest, and the SmokeTest will be changed to Lite version. Regarding STS, (Please check STS sheet in the attachment) We have new failure(android.security.sts.Poc18_05#testPocCVE_2017_18153), it's new item, because we update the security patch, we should also use new test tool. this new failure is also related to wifi, I think we can get waiver. Regarding GTS, (Please check GTS sheet in the attachment) We use new GTS tool this time, because the required date of new GTS (8.0_R2) is on 2021/1/15, There is one module can NOT be executed during the test, we have confirmed with Amlogic that this module is tools's bug. |
|
|
hi Jason.liang please change with v15.522.46 and start for a test. |
|
|
hi Jason.liang after changing with v17.522.48, please start a test. if you have some problem, please share it here. |
|
|
Hi Mr. Jason Park, We have finished 15.522.48. The result is the same as 15.522.42c. We have submited the report to 3PL. if they finished their job, I'll let you know. |
|
|
hi Jason.liang. if they finished their job, I'll let you know. --> what does 3PL do ? just upload a fxn's report ? |
|
|
Hi Mr. Jason Park, 3PL will help us upload the self test report and image for BTS scanning. They also create ticket and assign Google TAM to review it. Now, They finished uploading report. and we are waiting BTS result. if there is any update, I'll let you know. |
|
|
Hi Jason Ling I received V15.522.54 version from Jim Chen last night.(1/25) This is the final version for now, and SKB manager asked us to proceed with the Google certification test as soon as possible with this version. SKB wants to deliver this FW to the user immediately after the BMT test is over next week. It's a bit urgent right now, because STB users are reporting complaints Could you tell me when you can proceed with Google test with this latest version? If you let me know the available schedule (ASAP), I will talk to SKB manager. Thank you YK.Jung |
|
|
Hi Mr. Jung, Have you checked Youtube function in 15.522.54? I hear from Amlogic FAE that the Youtube function in 15.522.54 is fail. |
|
|
Hi Jason Ling I'm talking to Amlogic FAE. I will contact you after checking the issue. Thank you for the information. YK.Jung |
|
|
Hi Jason Ling Could you proceed with Google test with the FW released yesterday ( BFX-AT100 v15.523.6 ) ? personally , I would like to submit the test results to Google before March 23rd. Please test it as soon as possible. Thank you |
|
|
Hi Mr. Jung Update 15.523.6 status (2021/03/18, morning) CTS: In progress CTS-Verifier: In progress GTS: In progess VTS: finished, OK CTS-ON-GSI: In progress TVTS: finished, OK STS: finished, OK SmokeTest Lite: In progress Update 15.523.6 status (2021/03/18, night) CTS: In progress CTS-Verifier: OK GTS: OK VTS: finished, OK CTS-ON-GSI: In progress TVTS: finished, OK STS: finished, OK SmokeTest Lite: In progress |
|
|
Hi Jason Ling Thank you for updating the progress. SKB manager wants to know the progress of this Google certification. Please update the progress today morning as well Thank you YK.Jung |
|
|
Hi Mr. Jung Update 15.523.6 status (2021/03/19, morning) CTS: In progress (retrying, almost finished) CTS-Verifier: finished, OK GTS: finished, OK VTS: finished, OK CTS-ON-GSI: OK TVTS: finished, OK STS: finished, OK SmokeTest Lite: In progress (should be finished in this evening) How about BMT current status? |
|
|
Hi Jason I have a meeting with SKB manager at 11:00 AM ( korea time ) Please update the progress of Google before then. and if your Google test is completed without any issue, please submit it immediately. And please let me know what you submitted (link) If there's anything critical issue during BMT, I'll let you know right away. But not yet. Thank you YK.Jung |
|
|
Hi Mr. Jung Update 15.523.6 status (2021/03/22, morning) CTS: finished OK CTS-Verifier: finished, OK GTS: finished, OK VTS: finished, OK CTS-ON-GSI: OK TVTS: finished, OK STS: finished, OK SmokeTest Lite: OK Could you give us formal letter for submitting this version (15.523.6) to 3PL? |
|
|
Hi Jason I sent you an email. please check it Thanks |
|
|
Hi All, We have finished letter upgrade process from Android 9 to Android 10, If you have any certification request. please create new ticket. |
- yk.jung ( (ALTech) Younkwang Jung ) developer
